The invention is based on a priority application EP 02360192.5 which is hereby incorporated by reference.
This invention concerns a method of processing multimode user inputs, which are entered into a computer unit via various input means in various input modes within one specified application environment of several different ones, and converted there into different data streams.
The invention also concerns a multimode user interface, which processes multimode user inputs, which can be entered into a computer unit in various input modes within one specified application environment of several different ones, and converted there into different data streams.
Finally, this invention concerns a computer program for a multimode user interface, which can be run on a computing device, particularly a microprocessor, of the interface.
A method and user interface of the type mentioned above can be used, for instance, to control computer units, particularly PCs, mobile telephones and particularly to operate Internet-based or UMTS-based (Universal Mobile Telecommunications System) services. The user interface receives and processes user inputs which are entered by a user in a specifiable input mode, i.e. using various available input means such as a keyboard, a mouse, a tracker ball, a system for capturing and analysing hand-written input, a touch-sensitive screen, a speech recognition system, an optical system for capturing and analysing mimes and gestures of the user, or similar. In the computer unit, a data stream which depends on the type and content of the user input and the input modes which are used is generated.
The type of user input can be divided into different information categories. For instance, according to a broad division, the user input can be divided into the presentation, selection and execution information categories. In presentation, various services (selection options) are presented to the user optically, acoustically or otherwise. In selection, at least one of the selection options is selected, and information, particularly alphanumeric characters for texts, is input. For execution, a particular service, which has preferably previously been selected and supplied with information in the selection information category, is started and executed.
The user inputs can be generated within various application environments of a computer unit. According to this invention, the term “computer unit” is understood to include not only so-called PCs (personal computers), but any kind of electrical equipment which has a computing device, particularly a microprocessor, and which can be operated via user input. Thus, for the purposes of this invention, the term “computer unit” is understood to include a stationary or mobile telecommunication terminal. “Application environment” is understood to mean various application programs or parts of them, in which the user is during user input. In the case of a PC, for instance, an application environment for creating and editing text documents (text editor or text program) or an application environment for sending and receiving electronic messages (e-mail program) would be conceivable. In the case of telecommunication terminals, for instance, an application environment for writing, editing and receiving short messages (so-called SMS, Short Message Service) or an application environment for using telecommunication services, particularly WAP (Wireless Application Protocol) services, would be conceivable.
From WO 00/08547, a multimode user interface of the above-mentioned type and a method of processing the multimode user input of the above-mentioned type are known. However, this published specification is less about the processing sequence of the information categories within the individual data streams, but rather describes the type of processing of the individual data streams. In particular, it is proposed that a virtual machine (VM) correlates two or more different data streams with each other, and on the basis of the content of the two data streams which were processed last, determines an appropriate command for the computer unit. This command is then interpreted and executed in the computer unit, in a way which is known per se.
From the prior art, providing specified processing units (so-called interpreters) for service selection or service execution is also known. The user controls the change of interpreters, particularly which of the interpreters should be active for particular input modes and should interpret and/or execute commands irrespective of the information categories, through explicit instructions.
In the case of the method which is known from the state of the art for processing user input and the known multimode user interfaces, the strict delimitation between the various input modes for processing the individual data streams and the manual assignment of the interpreters by the user is disadvantageous. Thus, for instance, user input for executing a dialogue must be interpreted in a dialogue mode, and commands in a command mode. Parallel, flexibly controllable processing of multimode user input is impossible according to the prior art. In particular, it is impossible to control the processing by the computer unit fully automatically, by the user, depending on the context, i.e. the environment in which the user input was generated, and/or time-dependently.